Candidate answers for speculative questions in a deep question answering system

ABSTRACT

System, method, and computer program product to determine that a question received by a deep question answering system is speculative, generate, by one or more predictive algorithms, a set of candidate answers, compute a score for each candidate answer in the set of candidate answers, and return a first candidate answer, of the set of candidate answers, as responsive to the speculative question received by the deep question answering system.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of co-pending U.S. patent applicationSer. No. 14/177,624, filed Feb. 11, 2014. The aforementioned relatedpatent application is herein incorporated by reference in its entirety.

BACKGROUND

The present disclosure relates to computer software, and morespecifically, to computer software to dynamically generate candidateanswers for speculative questions in a deep question answering system.

Deep question answering systems answer questions by finding andevaluating candidate answers and supporting evidence from a corpus ofingested information. However, deep question answering systemstraditionally assume that answers and supporting evidence will exist ina known corpus. Therefore, deep question answering systems traditionallycannot answer “speculative” questions, i.e., those questions that do nothave an associated “answer” within the corpus.

SUMMARY

Embodiments disclosed herein include, without limitation, a system,method, and computer program product to determine that a questionreceived by a deep question answering system is speculative, generate,by one or more predictive algorithms, a set of candidate answers,compute a score for each candidate answer in the set of candidateanswers, and return a first candidate answer, of the set of candidateanswers, as responsive to the speculative question received by the deepquestion answering system.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

FIG. 1 depicts a system configured to dynamically generate candidateanswers for speculative questions in a deep question answering system,according to one embodiment.

FIG. 2 is a flow chart illustrating a method to dynamically generatecandidate answers for speculative questions in a deep question answeringsystem, according to one embodiment.

FIG. 3 is a flow chart illustrating a method to determine that aquestion is speculative, according to one embodiment.

FIG. 4 is a flow chart illustrating a method to generate candidateanswers for a speculative question, according to one embodiment.

FIG. 5 is a flow chart illustrating a method to set a trigger to verifyan answer to a speculative question, according to one embodiment.

DETAILED DESCRIPTION

Embodiments disclosed herein provide techniques to allow deep questionanswering (deep QA) systems to answer “speculative” questions asked byusers. As used herein, a “speculative” question refers to a questionthat, at the time presented by the user, does not have a definiteanswer. Examples of speculative questions include, without limitation,the price of an item (such as a commodity or precious metal) in thefuture (in two days, three weeks, four months, etc.), the winner of afuture sporting event, or which student will win the annual spelling beeat the local school.

In order to generate answers to speculative questions, the deep questionanswering system may be configured to determine that the question is infact speculative. Generally, the deep QA system many analyze thequestion text in order to detect a future tense or includes a concept(such as time) that is indicative of a speculative question. Upondetermining that the question is speculative, the deep QA system maythen generate new candidate answers based on insights gleaned from ananalysis of the speculative question and any associated evidencegathered from a corpus of information. Once the candidate answers aregenerated, the deep QA system may then score each candidate answer, andreturn the candidate answer with the highest score as being responsiveto the question.

Additionally, the deep QA system may subsequently evaluate the accuracyof each candidate answer and improve scoring algorithms after thequestion is no longer speculative, i.e., when an answer to the questionis known. When processing the speculative question, the deep QA systemmay store a trigger which causes the actual answer/outcome to bedetermined in the future, and compared against each candidate answer.For example, if the speculative question is “what will the price of cornbe in 2 months,” the deep QA system may set a trigger to fire in 2months, at which point the deep QA system may retrieve the actual priceof corn. The deep QA system may then compare the actual price of corn toeach candidate answer generated (and stored for future retrieval) atprocessing time. The trigger may be based on an elapsed period of time,or other criteria derived from the question, such as the performance orprice of an entity. The actual result/answer may be fed back into thedeep QA system in order to further improve the accuracy of the deep QAsystem as well as any predictive models used to generate the candidateanswers.

The deep QA system may determine that the question is speculative bycombining traditional question analysis (including concept and semanticrelation detection) along with verb tense annotators to identify afuture tense in the question.

FIG. 1 depicts a system 100 configured to dynamically generate candidateanswers for speculative questions in a deep question answering system,according to one embodiment. The networked system 100 includes acomputer 102. The computer 102 may also be connected to other computersvia a network 130. In general, the network 130 may be atelecommunications network and/or a wide area network (WAN). In aparticular embodiment, the network 130 is the Internet.

The computer 102 generally includes a processor 104 connected via a bus120 to a memory 106, a network interface device 118, a storage 108, aninput device 122, and an output device 124. The computer 102 isgenerally under the control of an operating system (not shown). Examplesof operating systems include the UNIX operating system, versions of theMicrosoft Windows operating system, and distributions of the Linuxoperating system. (UNIX is a registered trademark of The Open Group inthe United States and other countries. Microsoft and Windows aretrademarks of Microsoft Corporation in the United States, othercountries, or both. Linux is a registered trademark of Linus Torvalds inthe United States, other countries, or both.) More generally, anyoperating system supporting the functions disclosed herein may be used.The processor 104 is included to be representative of a single CPU,multiple CPUs, a single CPU having multiple processing cores, and thelike. Similarly, the memory 106 may be a random access memory. While thememory 106 is shown as a single identity, it should be understood thatthe memory 106 may comprise a plurality of modules, and that the memory106 may exist at multiple levels, from high speed registers and cachesto lower speed but larger DRAM chips. The network interface device 118may be any type of network communications device allowing the computer102 to communicate with other computers via the network 130.

The storage 108 may be a persistent storage device. Although the storage108 is shown as a single unit, the storage 108 may be a combination offixed and/or removable storage devices, such as fixed disc drives, solidstate drives, floppy disc drives, tape drives, removable memory cards oroptical storage. The memory 106 and the storage 108 may be part of onevirtual address space spanning multiple primary and secondary storagedevices.

The input device 122 may be any device for providing input to thecomputer 102. For example, a keyboard and/or a mouse may be used. Theoutput device 124 may be any device for providing output to a user ofthe computer 102. For example, the output device 124 may be anyconventional display screen or set of speakers. Although shownseparately from the input device 122, the output device 124 and inputdevice 122 may be combined. For example, a display screen with anintegrated touch-screen may be used.

As shown, the memory 106 contains the QA application 112, which is anapplication generally configured to operate a deep question answering(QA) system. One example of a deep question answering system is Watson,by the IBM Corporation of Armonk, N.Y. A user may submit a case (alsoreferred to as a question) to the QA application 112, which will thenprovide an answer to the case based on an analysis of a corpus ofinformation 114. The QA application 112 may analyze the questionspresented in the case to identify concepts in the question. Based on thequestions, the QA application 112 may identify a number of candidateanswers. The QA application 112 may then find supporting evidence forthe candidate answers. The QA application 112 may then score and rankthe candidate answers, merge the results, and present the best answer asits response to the case.

Additionally, the QA application 112 may be configured to answerspeculative questions submitted by a user. A speculative question is aquestion that when submitted by the user has an unknown answer. Forexample, the user may ask “who will win the race between the tortoiseand the hare?” Generally, the QA application 112 is configured to answerany type of speculative question. The speculative question need not betime-based, but lacks data that the QA application 112 can use togenerate a response to the question at the time the question isreceived. For example, a user may ask “how many unique people will haveregistered after 1,000,000 accounts are created on a new website?” Thisquestion does not need a specified amount of time to elapse prior toknowing the outcome, however, the data will not be in the corpus at thetime the user asks the question. However, if the corpus currentlyindicates that 7,000 unique people have registered for the first 10,000accounts on the website, the QA application 112 may use this data toproject an answer to the speculative question.

In order to answer the question, the QA application 112 may firstdetermine that the question is speculative. To determine that thequestion is speculative, the QA application 112 may apply traditionalquestion analysis techniques in the form of concept and semanticrelation detection combined with verb tense annotators in order toidentify a future tense in the question. Additionally, the QAapplication 112 may identify “independent variables” from the questionin order to hone in on the appropriate type of prediction. The“independent variables” may be concepts that include, withoutlimitation, time, prices, counts, populations, actions, and the like.Thus, the QA application 112 may understand whether or not the usermentioned a specific time (such as “in two months,” “this season,” “in2015”, etc.) together with the independent variables such as price orperformance of an entity. Therefore, the QA application 112 maydistinguish between whether the user is asking about the price of acommodity two months into the future versus two years into the future.

If the QA application 112 determines that a question is not speculative,the QA application 112 may process the question as usual. However, ifthe question is speculative, the QA application 112 may manufacture newcandidate answers based on information gleaned while determining whetherthe question is speculative. Generally, since the corpus 114 does notcontain the answers, the QA application 112 creates candidate answersvia inference by searching for evidence in the corpus related toattributes of the question (such as the price of oil on a specific date10 months in the future). The QA application 112 may use the attributesof the question to look up existing predictive algorithms in thepredictive algorithms 110 (or the corpus 114). If the predictivealgorithms stored in the predictive algorithms 110 need additional inputdata, the QA application 112 may gather the necessary data throughquestion analysis and associated evidence retrieval in order to pass theadditional input data to the selected algorithm.

The QA application 112 also has the ability to create new predictionalgorithms (or models) “on the fly” based on attributes from thequestion and evidence gathered by searching the corpus 114. In such ascenario, the QA application 112 may use attributes from the question inorder to gather and analyze past relevant evidence, looking for trendsin order to infer an answer. In one embodiment, the QA application 112only creates new prediction algorithms when no relevant predictionalgorithms exist in the predictive algorithms 110.

Once the QA application 112 generates one or more candidate answers (theoutput of the selected predictive algorithms from the predictivealgorithms 110), the QA application 112 scores each candidate answer,and returns the candidate answer with the highest score as beingresponsive to the question. In addition, the QA application 112 mayregister a trigger, stored in the triggers 117, which when fired, causesthe QA application 112 to conduct a “post-mortem” analysis of thecandidate answers generated in response to the speculative question. Forexample, if the user asks “will my favorite team win a championshipwithin the next two years,” the QA application 112 may register atrigger to fire at the expiration of the two year period (or toperiodically detect whether the team wins a championship before the twoyear period expires). Once the QA application 112 knows the answer, theQA application 112 may score each candidate answer generated by thedifferent predictive algorithms 110 used to answer the user's question.If the candidate answers generated by the algorithms are the correctanswer (or close to the correct answer within some threshold), the QAapplication 112 may assign a higher confidence score to future candidateanswers generated by the algorithms responsive to similar questions.Likewise, if the algorithms do not generate correct answers (or exceedan acceptable correctness threshold), the QA application 112 may assigna lower confidence score to future candidate answers generated by thealgorithms responsive to similar questions. In either scenario, theconfidence score may be for the algorithm, the candidate answersgenerated by the algorithms, or both. Additionally, the QA application112 may use the “correct answer” in order to retain its machine learningmodels 116 that are referenced to generate candidate answers throughtraditional evidence gathering processes and/or predictive algorithm 110selection decisions.

As shown, the storage 108 includes the predictive algorithms 110, acorpus 114, a machine learning (ML) models 116, and a triggers store117. The predictive algorithms 110 may include one or more predictivealgorithms (or models) that the QA application 112 may leverage in orderto generate candidate answers to speculative questions. The predictivealgorithms stored in the predictive algorithms 110 may be generated bythe QA application 112, or by a different source (such as specializedindustry-standard prediction models). For example, a predictivealgorithm in the predictive algorithms 110 may be an industry-standardmodel that predicts consumer spending or the United States grossdomestic product (GDP). The corpus 114 is a body of information used bythe QA application 112 to generate answers to questions (also referredto as cases). For example, the corpus 114 may contain scholarlyarticles, dictionary definitions, encyclopedia references, and the like.The corpus 114 typically does not include answers to speculativequestions, as the actual answers to these types of questions aretypically unknown. Machine learning (ML) models 116 are models createdby the QA application 112 during the training phase, which are usedduring a runtime pipeline to score and rank candidate answers to casesbased on features previously generated for each answer. The triggers 117include indications generated and stored by the QA application 112 totrigger a verification procedure by the QA application 112. The dataincluded in the triggers 117 may include, without limitation, atriggering time, event, or other attribute of each speculative question,along with the candidate answer values generated by each predictivealgorithm in the predictive algorithms 110.

FIG. 2 is a flow chart illustrating a method 200 to dynamically generatecandidate answers for speculative questions in a deep question answeringsystem, according to one embodiment. Generally, the steps of the method200 allow a deep question answering system to answer “speculative”questions, or questions that do not have a known answer at the timeasked by the user. In at least one embodiment, the QA application 112performs the steps of the method 200. At step 210, the QA application112 receives a question from a user. For example, the user may submit aquestion to the QA application 112 that asks “will it snow more than 5times in Florida during the year 2088?” At step 220, described ingreater detail with reference to FIG. 3, the QA application 112 maydetermine that the question received at step 210 is speculative.Generally, the QA application 112 may attempt to detect a future tensein the question text in order to determine that the question isspeculative.

At step 230, described in greater detail with reference to FIG. 4, theQA application 112 may generate candidate answers for the speculativequestion. At step 240, the QA application 112 may score the candidateanswers generated at step 240. In scoring the candidate answers, the QAapplication 112 may consider factors such as a confidence in thealgorithms used to generate candidate answers, a confidence in whetherthe QA application 112 correctly identified the question being asked(stated differently, whether the QA application 112 made an error ininterpreting the question), and a confidence in the machine learningmodels of the QA application 112. At step 250, the QA application 112may return the candidate answer receiving the highest score asresponsive to the question. At step 260, described in greater detailwith reference to FIG. 5, the QA application 112 may set a trigger towhich causes the QA application 112 to verify the answer to thespeculative question once the correct answer is available. In verifyingthe answer, the QA application 112 may compare each candidate answergenerated at step 230 to the correct answer. The QA application 112 mayuse the comparison results to update a confidence score in thepredictive algorithms 110 generating the candidate answer, as well as toretrain the ML models 116.

FIG. 3 is a flow chart illustrating a method 300 corresponding to step220 to determine that a question is speculative, according to oneembodiment. In at least one embodiment, the QA application 112 performsthe steps of the method 300. As previously indicated, a speculativequestion is a question that does not have a definite correct answer whenasked by the user. Stated differently, in determining whether a questionis speculative, the QA application 112 determines how likely it is thatthe question has an answer in the corpus 114. At step 310, the QAapplication 112 performs an analysis of the question text. In at leastsome embodiments, the analysis includes an analysis of the question textthat is traditionally performed by the QA application 112 on allquestions. For example, the traditional analysis may include, withoutlimitation, concept detection, semantic relation detection, and verbtense annotators.

At step 320, the QA application 112 determines whether the future tenseexists in the question. For example, if the question is “what will theprice of corn be in 2 months,” the QA application 112, using verb tenseannotators, may identify the future tense in the portion of the questionthat reads “in 2 months.” If the QA application 112 detects the futuretense in the question, it is a strong indicator that the question isspeculative. At step 330, the QA application 112 determines whetherspeculative independent variables exist in the question. The independentvariables may include, without limitation, time, prices, counts,populations, and the like. In identifying independent variables, the QAapplication 112 may be more certain that the question is speculative.Furthermore, the independent variables may assist the QA application 112in identifying the correct thrust of the question, such that the QAapplication 112 understands the type of prediction the question isasking for. Therefore, by identifying independent variables in theexample above, the QA application 112 may be more certain that the userwants to know the price of corn in two months, and the QA application112 may focus its search for supporting evidence in the corpus 114 basedon this concept. At step 340, the QA application 112 determines whetherspeculative concepts exist in the question. For example, if the questionasks “will it rain in New York,” the QA application 112 may determinethat the user wants to know what the weather will be like, which isnecessarily a speculative concept, as the user is asking about weatherin the future, even though the user did not specify a specific date ortime. If the QA application 112 detects speculative concepts in thequestion, it may be more likely that the question is indeed speculative.

At step 350, the QA application 112 scores the question to determinewhether the question is speculative. For example, the QA application 112may weight one or more factors in computing a score for the question. Ifthe score exceeds a speculative threshold, the QA application 112 maydetermine that the question is speculative. The QA application 112 mayuse, without limitation, one or more of the future tense, speculativeconcepts, and independent variables detected steps 320-340 in order tocompute the score. The QA application 112 may weigh each of theseattributes differently in order to score the question. If the QAapplication 112 determines that the question is speculative, the QAapplication 112 may inject additional candidate answers to its questionanswering pipeline, in addition to one or more existing non-speculativecandidate answers to the question.

In addition, the QA application 112 may modify the scoring of candidateanswers by a speculative score for the question. For example, if the QAapplication 112 is 20% confident that the question speculative, but 100%confident in a candidate answer to the speculative question, the QAapplication 112 may score the candidate answer as 20% confident due tothe uncertainty that the question is speculative. As another example,the QA application 112 may be 100% confident that another question isspeculative and 50% confident in the candidate answer to the question.In such a case, the QA application 112 may be 50% confident in thecandidate answer. Generally, the QA application 112 may use anyalgorithm sufficient to account for its confidence level that thequestion is speculative when scoring candidate answers, including,without limitation, multiplying the confidence scores that the questionis speculative and the candidate answer is correct.

FIG. 4 is a flow chart illustrating a method 400 corresponding to step230 to generate candidate answers for a speculative question, accordingto one embodiment. In at least some embodiments, the QA application 112executes the steps of the method 400. Because the QA application 112 hasdetermined the question is speculative, and an answer is not included inthe corpus 114, the QA application 112 executes the steps of the method400 in order to dynamically create candidate answers through inference.At step 410, the QA application 112 gathers evidence from the corpus 114related to the key attributes, variables and concepts identified in thequestion. At step 420, the QA application 112 searches the predictivealgorithms 110 (and possibly the corpus 114) in order to identifyexisting predictive algorithms. For example, if the question asks theprice of a thousand widgets in two months, the QA application 112 maysearch for algorithms that predict commodity prices, including widgetprices, if such an algorithm is available in the predictive algorithms110. The predictive algorithms 110 may be mostly self-contained in thatone or more algorithms may be called with minimal additional input. Forexample, an algorithm for predicting future widget prices may know howto gather all the data it needs to generate an answer with only a smallamount of input, such as a target date. Other algorithms in thepredictive algorithms 110 may require input data that cannot beautomatically retrieved. In such cases, the QA application 112 maycompile the necessary input data through question analysis andassociated evidence retrieval.

At step 430, the QA application 112 may optionally create predictivealgorithms “on the fly” based on attributes from the question andevidence gathered from searching the corpus 114. The QA application 112may concurrently generate the predictive algorithms while finding andinvoking the existing algorithms from the predictive algorithms 110 insteps 420-430. The algorithms generated by the QA application 112 may becompletely self-contained within the QA application 112, requiringlittle to no additional input data. The QA application 112 may useattributes from the question in order to gather and analyze pastrelevant evidence from the corpus 114, looking for trends in order toinfer an answer. The algorithms generated by the QA application 112provide a general purpose way of generating a candidate answer in theabsence of any existing prediction algorithms relevant to the questionin the predictive algorithms 110. Once generated, the QA application 112may store the new algorithm in the predictive algorithms 110. At step440, the QA application 112 may pass the required inputs and invoke theone or more of the predictive algorithms 110 in order to generatecandidate answers. The candidate answers generated by each of thealgorithms (existing or generated by the QA application 112) may betailored to the question. For example, a first algorithm may return acandidate answer indicating that the price of one thousand widgets in 2months will be $10, while a second algorithm may return a price of $11,and a third algorithm may return a candidate answer indicating that theprice of one thousand widgets will rise by 5% relative to currentprices. Generally, the candidate answer may take any form sufficient tobe responsive to the speculative question.

FIG. 5 is a flow chart illustrating a method 500 corresponding to step260 to set a trigger to verify an answer to a speculative question,according to one embodiment. In at least some embodiments, the QAapplication 112 performs the steps of the method 500. Generally, thesteps of the method 500 allow the QA application 112 to verify whetherpreviously generated candidate answers were accurate, adjust confidencescores in the predictive algorithms 110 generating the candidateanswers, and adjust confidence scores in its own machine learning models116. At step 510, the QA application 112 identifies a time periodrelevant to the speculative question and stores a trigger in thetriggers 117. For example, if the question asks the price of corn in 6months, the trigger may be set to fire in 6 months. However, the triggerneed not be time based. For example, and without limitation, if the userwishes to know how many unique users correspond to the first millionaccounts on a website, the QA application 112 may set a trigger tomonitor the website's account creation statistics. The QA application112 may then periodically check the website account creation statisticsuntil the website reaches one million created accounts. When storing thetrigger, the QA application 112 may also store the different candidateanswers for the question generated by the predictive algorithms as wellas the QA application 112 for future comparison. At step 520, the timeperiod elapses (or other condition is met), and the trigger is fired. Atstep 530, the QA application 112 identifies the correct answer oroutcome of the question. For example, the QA application 112 mayretrieve the current price of corn, or how many unique users createdaccounts out of the first million accounts on the website.

At step 540, the QA application 112 executes a loop including steps550-560 for each predictive algorithm that generated a candidate answer.At step 550, the QA application 112 adjusts the confidence score of thecurrent predictive algorithm based on the accuracy of the candidateanswer generated by the predictive algorithm. The QA application 112 mayuse any feasible method to modify the confidence score of the predictivealgorithm. For example, the QA application 112 may allocate additionalconfidence to the predictive algorithm if it predicted the correctanswer (or the predicted answer was within a specified confidencethreshold). Conversely, if the predictive algorithm did not predict thecorrect answer, or was in error beyond a reasonable threshold, the QAapplication 112 may reduce the confidence score of the predictivealgorithm. In another embodiment, the QA application 112 may ranks thealgorithms that produce the answer closest to the correct answer morehighly, such that when the ML models 116 are retrained, they areconsidered more accurately predictive for future similar questions. Atstep 560, the QA application 112 determines whether more predictivealgorithms remain. If more predictive algorithms remain, the QAapplication 112 returns to step 540. If no predictive algorithms remain,the QA application 112 proceeds to step 570, where the QA application112 retrains the ML models 116. By retraining the ML models 116, the QAapplication 112 trains itself to make more accurate decisions onprocessing similar speculative questions in the future.

Advantageously, embodiments disclosed herein provide a deep questionanswering system the ability to answer speculative questions, eventhough an answer to the speculative questions are not found in a corpusof information traditionally used by the system to answer questions.Generally, the deep question answering system may identify a receivedquestion as speculative by detecting a future tense or other indicatorof speculation. The deep question answering system may then generatecandidate answers to the speculative questions based on one or morepre-existing predictive algorithms. The deep question answering systemmay also create one or more predictive algorithms to generate candidateanswers in addition to (or in lieu of) the existing predictivealgorithms. The deep question answering system may then score eachcandidate answer, and return the highest scoring candidate answer asresponsive to the speculative question. In addition, the deep questionanswering system may conduct a subsequent verification of the correctanswer to the speculative question by comparing the correct answer tothe generated candidate answers at a later time. The verificationprocess may allow the deep question answering system to adjustconfidence scores in the predictive algorithms and retrain its ownmachine learning models.

The descriptions of the various embodiments of the present disclosurehave been presented for purposes of illustration, but are not intendedto be exhaustive or limited to the embodiments disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the describedembodiments. The terminology used herein was chosen to best explain theprinciples of the embodiments, the practical application or technicalimprovement over technologies found in the marketplace, or to enableothers of ordinary skill in the art to understand the embodimentsdisclosed herein.

As will be appreciated by one skilled in the art, aspects of the presentdisclosure may be embodied as a system, method or computer programproduct. Accordingly, aspects of the present disclosure may take theform of an entirely hardware embodiment, an entirely software embodiment(including firmware, resident software, micro-code, etc.) or anembodiment combining software and hardware aspects that may allgenerally be referred to herein as a “circuit,” “module” or “system.”Furthermore, aspects of the present disclosure may take the form of acomputer program product embodied in one or more computer readablemedium(s) having computer readable program code embodied thereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readable signalmedium or a computer readable storage medium. A computer readablestorage medium may be, for example, but not limited to, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, or device, or any suitable combination of the foregoing. Morespecific examples (a non-exhaustive list) of the computer readablestorage medium would include the following: an electrical connectionhaving one or more wires, a portable computer diskette, a hard disk, arandom access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or Flash memory), an optical fiber,a portable compact disc read-only memory (CD-ROM), an optical storagedevice, a magnetic storage device, or any suitable combination of theforegoing. In the context of this document, a computer readable storagemedium may be any tangible medium that can contain, or store a programfor use by or in connection with an instruction execution system,apparatus, or device.

A computer readable signal medium may include a propagated data signalwith computer readable program code embodied therein, for example, inbaseband or as part of a carrier wave. Such a propagated signal may takeany of a variety of forms, including, but not limited to,electro-magnetic, optical, or any suitable combination thereof. Acomputer readable signal medium may be any computer readable medium thatis not a computer readable storage medium and that can communicate,propagate, or transport a program for use by or in connection with aninstruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmittedusing any appropriate medium, including but not limited to wireless,wireline, optical fiber cable, RF, etc., or any suitable combination ofthe foregoing.

Computer program code for carrying out operations for aspects of thepresent disclosure may be written in any combination of one or moreprogramming languages, including an object oriented programming languagesuch as Java, Smalltalk, C++ or the like and conventional proceduralprogramming languages, such as the “C” programming language or similarprogramming languages. The program code may execute entirely on theuser's computer, partly on the user's computer, as a stand-alonesoftware package, partly on the user's computer and partly on a remotecomputer or entirely on the remote computer or server. In the latterscenario, the remote computer may be connected to the user's computerthrough any type of network, including a local area network (LAN) or awide area network (WAN), or the connection may be made to an externalcomputer (for example, through the Internet using an Internet ServiceProvider).

Aspects of the present disclosure are described below with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems) and computer program products according to embodiments of thedisclosure. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, create means for implementing the functions/acts specified inthe flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a computerreadable medium that can direct a computer, other programmable dataprocessing apparatus, or other devices to function in a particularmanner, such that the instructions stored in the computer readablemedium produce an article of manufacture including instructions whichimplement the function/act specified in the flowchart and/or blockdiagram block or blocks.

The computer program instructions may also be loaded onto a computer,other programmable data processing apparatus, or other devices to causea series of operational steps to be performed on the computer, otherprogrammable apparatus or other devices to produce a computerimplemented process such that the instructions which execute on thecomputer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

Embodiments of the disclosure may be provided to end users through acloud computing infrastructure. Cloud computing generally refers to theprovision of scalable computing resources as a service over a network.More formally, cloud computing may be defined as a computing capabilitythat provides an abstraction between the computing resource and itsunderlying technical architecture (e.g., servers, storage, networks),enabling convenient, on-demand network access to a shared pool ofconfigurable computing resources that can be rapidly provisioned andreleased with minimal management effort or service provider interaction.Thus, cloud computing allows a user to access virtual computingresources (e.g., storage, data, applications, and even completevirtualized computing systems) in “the cloud,” without regard for theunderlying physical systems (or locations of those systems) used toprovide the computing resources.

Typically, cloud computing resources are provided to a user on apay-per-use basis, where users are charged only for the computingresources actually used (e.g. an amount of storage space consumed by auser or a number of virtualized systems instantiated by the user). Auser can access any of the resources that reside in the cloud at anytime, and from anywhere across the Internet. In context of the presentdisclosure, a user may access applications or related data available inthe cloud. For example, the QA application 112 could execute on acomputing system in the cloud and answer speculative questions. In sucha case, the QA application 112 could generate candidate answers andstore the candidate answers and a verification trigger at a storagelocation in the cloud. Doing so allows a user to access this informationfrom any computing system attached to a network connected to the cloud(e.g., the Internet).

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods and computer program products according to variousembodiments of the present disclosure. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof code, which comprises one or more executable instructions forimplementing the specified logical function(s). It should also be notedthat, in some alternative implementations, the functions noted in theblock may occur out of the order noted in the figures. For example, twoblocks shown in succession may, in fact, be executed substantiallyconcurrently, or the blocks may sometimes be executed in the reverseorder, depending upon the functionality involved. It will also be notedthat each block of the block diagrams and/or flowchart illustration, andcombinations of blocks in the block diagrams and/or flowchartillustration, can be implemented by special purpose hardware-basedsystems that perform the specified functions or acts, or combinations ofspecial purpose hardware and computer instructions.

While the foregoing is directed to embodiments of the presentdisclosure, other and further embodiments of the disclosure may bedevised without departing from the basic scope thereof, and the scopethereof is determined by the claims that follow.

What is claimed is:
 1. A method, comprising: determining that a questionreceived by a deep question answering system is speculative; generating,by one or more predictive algorithms, a set of candidate answers;computing a score for each candidate answer in the set of candidateanswers; and returning a first candidate answer, of the set of candidateanswers, as responsive to the speculative question received by the deepquestion answering system.
 2. The method of claim 1, further comprising:storing an indication specifying to verify each of the candidate answerssubsequent to a time related to the question; and upon determining thatthe time has passed: comparing each candidate answer in the set ofcandidate answers to an actual answer to the question to determine anaccuracy of each candidate answer; adjusting a confidence score of anability of the deep question answering system to generate the set ofcandidate answers; and adjusting a confidence score of the predictivealgorithm that generated the candidate answer in the set of candidateanswers based on the accuracy of each respective candidate answer. 3.The method of claim 1, wherein determining that the question isspeculative comprises at least one of: (i) identifying a future tense inthe question, (ii) identifying an independent variable in the question,and (iii) determining that a corpus of information does not include acandidate answer for the question.
 4. The method of claim 1, whereingenerating the set of candidate answers comprises: collecting relevantevidence from a corpus of information; selecting the one or morepredictive algorithms; and applying the one or more predictivealgorithms to the relevant evidence and at least one attribute of thequestion.
 5. The method of claim 1, further comprising: computing aconfidence score of an analysis of the question and supporting evidencegathered from a corpus of information based on the analysis of thequestion; and adjusting the score of each candidate answer based on thecomputed confidence score.
 6. The method of claim 1, wherein the one ormore predictive algorithms are selected from: (i) a set of existingpredictive algorithms, and (ii) one or more predictive algorithmsgenerated by the deep question answering system.
 7. The method of claim6, wherein the predictive algorithms generated by the deep questionanswering system are based on: (i) at least one attribute of thequestion, (ii) relevant evidence gathered from a corpus of information,(iii) one or more trends found in the corpus of information.